Tag: KV cache
3 reviews
The LLM inference engine playbook: tricks in vLLM and SGLang
Which optimization techniques do the vLLM and SGLang serving engines implement, and in what order should an engineer study them?
LLM serving engines are built from a stack of optimization tricks, each attacking one bottleneck: batching keeps the GPU busy, KV-cache paging and reuse stop memory from fragmenting or being recomputed, faster attention kernels and lower-precision arithmetic cut bytes moved, speculative decoding trades spare compute for fewer serial steps, and prefill-decode disaggregation lets two workloads with opposite resource profiles run on machines tuned for each. This review catalogs the trick list of the two dominant open-source engines, vLLM and SGLang, maps each trick to the paper that introduced it and the bottleneck it addresses, and closes with an ordered self-study path. Confidence is moderate: the mechanisms are well established in the literature, but the engines' own performance claims are vendor-reported and rarely independently benchmarked.
Optimizing large language model inference
Which techniques most improve the throughput, latency, and cost of large language model inference?
Large language model inference is memory-bound, not compute-bound, and every major optimization family attacks the memory wall from a different angle: batching and scheduling to keep GPUs busy, KV-cache management to shrink the state that grows with context, quantization to cut bytes per weight, speculative decoding to trade spare compute for fewer serial steps, and prefill-decode separation to stop two workloads with opposite resource profiles from interfering. The evidence is consistent that these techniques deliver large gains — commonly 2–6x in throughput or latency, occasionally 10x or more in narrow regimes — but nearly every headline number is an "up to" figure measured on the authors' own hardware and workload, and independent benchmarks are only beginning to appear.
Interconnects for large-scale LLM training and inference: RDMA, InfiniBand, NVLink, and the network stack
What does the evidence from 2020 to 2026 show about how interconnect technologies (RDMA, InfiniBand, NVLink, Ethernet/RoCE, and the software stacks built on them) shape the performance, scaling, and cost of large-scale LLM training and inference?
Large-scale LLM training and inference now depend on a hierarchy of interconnects: NVLink and NVSwitch inside servers, InfiniBand or RoCE Ethernet between them, and collective-communication libraries that turn raw fabric into training progress. The 2020-2026 evidence shows the lossless RDMA fabric that carried early GPT-era training is under strain, with congestion-control pathologies and head-of-line blocking documented in both directions, while production operators engineer around it and a lossy, packet-spraying Ultra Ethernet path is emerging. Meanwhile Mixture-of-Experts models and long-context inference have made all-to-all traffic and KV-cache movement the new communication bottlenecks, and NVLink scale-up domains, UALink, and CXL memory pooling are reshaping where computation and memory live. Confidence is moderate: the transport and collective stories are replicated across many independent studies, but the newest hardware (NVLink 4/5, UALink, GB200-class superpods) is covered mainly by vendor-adjacent or preprint evidence.